Field of the Invention
One or more embodiments of the invention are related to the field of image analysis and image enhancement and enable the rapid conversion of a sequence of images, for example a movie. More particularly, but not by way of limitation embodiments enable conversion of black and white images to color images and/or two-dimensional images into three-dimensional images based on adding color and/or depth to images using masks for regions in the images, as well as reshaping of masks to cover objects that have moved and changed shape as the objects move in a sequence of images. In addition, embodiments also are related to the field of project management in the motion picture industry and relates to reviewers, production managers who manage artists. Production managers are also known as “production” for short. Artists utilize image analysis and image enhancement and computer graphics processing for example to convert two-dimensional images into three-dimensional images associated with a motion picture or otherwise create or alter motion pictures. More particularly, but not by way of limitation, one or more embodiments of the invention enable motion picture project management system configured to efficiently manage projects related to motion pictures to manage assets, control costs, predict budgets and profit margins, reduce archival storage and otherwise provide displays tailored to specific roles to increase worker efficiency.
Description of the Related Art
Known methods for the colorizing of black and white feature films involves the identification of gray scale regions within a picture followed by the application of a pre-selected color transform or lookup tables for the gray scale within each region defined by a masking operation covering the extent of each selected region and the subsequent application of said masked regions from one frame to many subsequent frames. The primary difference between U.S. Pat. No. 4,984,072, System And Method For Color Image Enhancement, and U.S. Pat. No. 3,705,762, Method For Converting Black-And-White Films To Color Films, is the manner by which the regions of interest (ROIs) are isolated and masked, how that information is transferred to subsequent frames and how that mask information is modified to conform with changes in the underlying image data. In the U.S. Pat. No. 4,984,072 system, the region is masked by an operator via a one-bit painted overlay and operator manipulated using a digital paintbrush method frame by frame to match the movement. In the U.S. Pat. No. 3,705,762 process, each region is outlined or rotoscoped by an operator using vector polygons, which are then adjusted frame by frame by the operator, to create animated masked ROIs. Various masking technologies are generally also utilized in the conversion of 2D movies to 3D movies.
In both systems described above, the color transform lookup tables and regions selected are applied and modified manually to each frame in succession to compensate for changes in the image data that the operator detects visually. All changes and movement of the underlying luminance gray scale is subjectively detected by the operator and the masks are sequentially corrected manually by the use of an interface device such as a mouse for moving or adjusting mask shapes to compensate for the detected movement. In all cases the underlying gray scale is a passive recipient of the mask containing pre-selected color transforms with all modifications of the mask under operator detection and modification. In these prior inventions the mask information does not contain any information specific to the underlying luminance gray scale and therefore no automatic position and shape correction of the mask to correspond with image feature displacement and distortion from one frame to another is possible.
Existing systems that are utilized to convert two-dimensional images to three-dimensional images may also require the creation of wire frame models for objects in images that define the 3D shape of the masked objects. The creation of wire frame models is a large undertaking in terms of labor. These systems also do not utilize the underlying luminance gray scale of objects in the images to automatically position and correct the shape of the masks of the objects to correspond with image feature displacement and distortion from one frame to another. Hence, great amounts of labor are required to manually shape and reshape masks for applying depth or Z-dimension data to the objects. Motion objects that move from frame to frame thus require a great deal of human intervention. In addition, there are no known solutions for enhancing two-dimensional images into three-dimensional images that utilize composite backgrounds of multiple images in a frame for spreading depth information to background and masked objects. This includes data from background objects whether or not pre-existing or generated for an occluded area where missing data exists, i.e., where motion objects never uncover the background. In other words, known systems gap fill using algorithms for inserting image data where none exists, which causes artifacts.
Current methods for converting movies from 2D to 3D that include computer-generated elements or effects, generally utilize only the final sequence of 2D images that make up the movie. This is the current method used for conversion of all movies from two-dimensional data to left and right image pairs for three-dimensional viewing. There are no known current methods that obtain and make use of metadata associated with the computer-generated elements for a movie to be converted. This is the case since studios that own the older 2D movies may not have retained intermediate data for a movie, i.e., the metadata associated with computer generated elements, since the amount of data in the past was so large that the studios would only retain the final movie data with rendered computer graphics elements and discard the metadata. For movies having associated metadata that has been retained, (i.e., intermediate data associated with the computer-generated elements such as mask, or alpha and/or depth information), use of this metadata would greatly speed the depth conversion process.
In addition, typical methods for converting movies from 2D to 3D in an industrial setting capable of handling the conversion of hundreds of thousands of frames of a movie with large amounts of labor or computing power, make use of an iterative workflow. The iterative workflow includes masking objects in each frame, adding depth and then rendering the frame into left and right viewpoints forming an anaglyph image or a left and right image pair. If there are errors in the edges of the masked objects for example, then the typical workflow involves an “iteration”, i.e., sending the frames back to the workgroup responsible for masking the objects, (which can be in a country with cheap unskilled labor half way around the world), after which the masks are sent to the workgroup responsible for rendering the images, (again potentially in another country), after which the rendered image pair is sent back to the quality assurance group. It is not uncommon in this workflow environment for many iterations of a complicated frame to take place. This is known as “throw it over the fence” workflow since different workgroups work independently to minimize their current work load and not as a team with overall efficiency in mind. With hundreds of thousands of frames in a movie, the amount of time that it takes to iterate back through frames containing artifacts can become high, causing delays in the overall project. Even if the re-rendering process takes place locally, the amount of time to re-render or ray-trace all of the images of a scene can cause significant processing and hence delays on the order of at least hours. Elimination of iterations such as this would provide a huge savings in wall-time, or end-to-end time that a conversion project takes, thereby increasing profits and minimizing the workforce needed to implement the workflow.
General simplistic project management concepts are known, however the formal and systematic application of project management in engineering projects of large complexity began in the mid-1900's. Project management in general involves at least planning and managing resources and workers to complete a temporary activity known as a project. Projects are generally time oriented and also constrained by scope and budget. Project management was first described in a systematic manner by Frederick Winslow Taylor and his students Henry Gantt and Henri Fayol. Work breakdown structure and Gantt charts were used initially and Critical Path Method “CPM” and Program Evaluation and Review Technique “PERT” were later developed in industrial and defense settings respectively. Project cost estimating followed these developments. Basic project management generally includes initiation, project planning, execution, monitor/control and completion. More complex project management techniques may attempt to achieve other goals, such as ensuring that the management process is defined, quantitatively managed and optimized for example as is described in the Capability Maturity Model Integration approach.
As described above, industrial based motion picture projects typically include hundreds of thousands of frames, however in addition, these types of projects may also utilize use gigantic amounts of storage including potentially hundreds of layers of masks and images per frame and hundreds of workers. These types of projects have been managed in a fairly ad hoc manner to date in which costs are difficult to predict, controlled feedback to redirect a project toward financial success, asset management and most other best practice project management techniques are minimally utilized. In addition, project management tools utilized include off the shelf project management tools that are not tailored for the specifics of project management in a unique vertical industry such as motion picture effects and conversion projects. Hence, predicting costs and quality and repeatedly performing projects in the film industry has been difficult to accomplish to date. For example, existing motion picture projects sometimes require three people to review an edited frame in some cases, e.g., a person to locate the resource amongst a large number of resources, a person to review the resource and another person to provide annotations for feedback and rework. Although standalone tools exist to perform these tasks, they are generally not integrated and are difficult for personnel in different roles to utilize.
Regardless of the known techniques, there are no known optimizations or implementations of project management solutions that take into account the unique requirements of the motion picture industry. Hence there is a need for a motion picture project management system.